-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-46586: Modify embargo-butler auto-ingest to handle photodiode files #60
base: main
Are you sure you want to change the base?
Conversation
LSSTCam is the only instrument with photodiode data that may need to be ingested. An example of the photodiode file path is rubinobs-lfa-cp/MTCamera/photodiode/20240928/MC_C_20240928_002826_photodiode.ecsv
2986a4e
to
fb4f951
Compare
@@ -8,14 +8,14 @@ and automatically ingest them into an appropriate Butler repository. | |||
Containers | |||
---------- | |||
|
|||
Three containers are built from this repo: enqueue, ingest, and idle. | |||
Four containers are built from this repo: enqueue, ingest, idle, and presence. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks :)
@@ -146,9 +146,12 @@ def __init__(self, path): | |||
self.instrument = f"{csc}/{generator}" | |||
elif len(components) == 6: | |||
self.bucket, self.instrument, year, month, day, self.filename = components | |||
elif len(components) == 5: # photodiode data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm. This really says that the Camera is not following salobj precedent in generating its LFA data. I guess it may be too late to get that changed.
@@ -220,6 +234,13 @@ def main(): | |||
|
|||
logger.info("Ingesting %s", resources) | |||
refs = None | |||
if is_lfa: | |||
resources_photodiode = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Internet says you could write this as:
resources_photodiode = []
resources = []
for resource in resources:
(resources_photodiode if re.search(r"MTCamera/photodiode.*_photodiode.ecsv$", resource) else resources).append(resource)
Even the fully explicit:
for resource in resources:
if re.search():
resources_photodiode.append(resource)
else:
resources.append(resource)
might be clearer.
@@ -233,6 +254,22 @@ def main(): | |||
info = Info.from_path(resource.geturl()) | |||
r.lrem(worker_queue, 0, info.path) | |||
|
|||
if is_lfa and resources_photodiode: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is the same structure as the previous section, it might be nice to abstract that out into a function, in case we have more ingesters later.
I don't like so many special casing, but it might be necessary unless
RawIngestTask
is made to accept photodiode data?